Using the Equivalent Kernel to Understand Gaussian Process Regression

The equivalent kernel [1] is a way of understanding how Gaussian process regression works for large sample sizes based on a continuum limit. In this paper we show (1) how to approximate the equivalent kernel of the widely-used squared exponential (or Gaussian) kernel and related kernels, and (2) how analysis using the equivalent kernel helps to understand the learning curves for Gaussian processes. Consider the supervised regression problem for a dataset D with entries (xi, yi) for i = 1, . . . , n. Under Gaussian Process (GP) assumptions the predictive mean at a test point x∗ is given by f̄(x∗) = k (x∗)(K + σ I)y, (1) where K denotes the n × n matrix of covariances between the training points with entries k(xi,xj), k(x∗) is the vector of covariances k(xi,x∗), σ is the noise variance on the observations and y is a n × 1 vector holding the training targets. See e.g. [2] for further details. We can define a vector of functions h(x∗) = (K + σI)k(x∗) . Thus we have f̄(x∗) = h(x∗)y, making it clear that the mean prediction at a point x∗ is a linear combination of the target values y. Gaussian process regression is thus a linear smoother, see [3, section 2.8] for further details. For a fixed test point x∗, h(x∗) gives the vector of weights applied to targets y. Silverman [1] called h(x∗) the weight function. Understanding the form of the weight function is made complicated by the matrix inversion of K + σI and the fact that K depends on the specific locations of the n datapoints. Idealizing the situation one can consider the observations to be “smeared out” in x-space at some constant density of observations. In this case analytic tools can be brought to bear on the problem, as shown below. By analogy to kernel smoothing Silverman [1] called the idealized weight function the equivalent kernel (EK). The structure of the remainder of the paper is as follows: In section 1 we describe how to derive the equivalent kernel in Fourier space. Section 2 derives approximations for the EK for the squared exponential and other kernels. In section 3 we show how use the EK approach to estimate learning curves for GP regression, and compare GP regression to kernel regression using the EK. 1 Gaussian Process Regression and the Equivalent Kernel It is well known (see e.g. [4]) that the posterior mean for GP regression can be obtained as the function which minimizes the functional J [f ] = 1 2 ‖f‖H + 1 2σ2 n n